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(57) Abstract 

A system and method providing for the generation of an image having enhanced three-dimensional attributes from a source image 
having only two-dimensional spatial domain information is disclosed. Object image discrimination through the use of machine vision 
techniques is utilized to identify objects within the source image. Thereafter, object images are dissected into image cells wherein the object 
images are supplemented to includes missing information. Generation techniques are utilized to reassemble the image cells into an image 
having the desired enhanced three dimensional attributes. 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


SZ 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


uz 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


Zimbabwe 


CI 


C6te d'lvoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






CU 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






CZ 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







WO 00/30039 



1 



PCT/US99/27607 



SYSTEM AND METHOD FOR CREATING 3D MODELS FROM 2D 
SEQUENTIAL IMAGE DATA 

TECHNICAL FIELD OF THE INVENTION 

The present invention relates generally to digital manipulation of image signals and 
more specifically to a system and method for creating 3D models from 2D digital input 
images. 
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BACKGROUND OF THE INVENTION 

The depiction of visual images as spatial domain representations displayed by a system 
on a suitable display device, such as the cathode ray tube ("CRT") of a television or computer 
monitor or via film projected on a screen, is well known in the art. However, as such display 
5 devices are typically limited to presenting the representations as a two-dimensional spacial 

representation on the surface of the display device, these spatial domain representations do not 
include complete visual image information. For example, due to occlusion by a foreground 
object, image information with respect to a background object may be missing. Likewise, 
image depth, or parallax, information, discernable as an apparent displacement in the position 
10 of an object as seen against a more distant background or other object when the viewpoint is 

changed, is typically lost in two-dimensional representations. 

However, acquisition of such visual images to be represented two-dimensionally is 
straight forward, using opto-electronic transducing devices such as movie film cameras, video 
cameras and computer scanning devices to capture a spatial domain representation from a 

15 single vantage point, where the source of the visual image is a perceptible image. Likewise, 

acquisition of two-dimensional images generated on a digital system using a variety of 
computer software programs, such as word processing, drawing and animation programs, 
where the source of the visual image is imperceptible is also straight forward. Therefore, there 
is a wealth of video images and film images captured and stored as two-dimensional 

20 representations as well as infrastructure, including systems and equipment, for such two- 
dimensional image acquisition. 

Regardless of their source, these acquired images may be represented and stored in 
digital systems as arrays of digital numbers. A digitized image is simply a set of numbers 
having a direct correspondence with the pixels of the displayed image. For example, a 
25 displayed image might consist of 512 by 640 pixels where each pixel is characterized by a 

range of possible luminous intensities and colors. Film images can also be processed into a 
matrix of pixels similar to video images. 

Processing of digital video images is well known in the art. Traditionally, such prior 
art digital video processing has been divisible into two major categories. The first prior art 
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category results in a new video image being produced such as through the use of chroma- 
keying, image compositing and overlaying, rendering, transitions including wipes and fades, 
and computer generated images including three dimensional computer models and titles. 
These techniques and their like may be generally categorized as "video generation" 
techniques, and result in a new two-dimensional spatial domain representation of a video 
image. 

Contrariwise the second prior art category processes a video image not to generate a 
new video image, but rather to discern information therefrom, such as in an effort to recognize 
objects from within the image. Such processing is often used in robotics, for example, in 
order for feedback to be provided with respect to the operation of a manipulator. Video 
processing to differentiate between objects appearing in the image or to otherwise discern 
information contained within the image may be generally categorized as "machine vision" 
techniques. 

It shall be appreciated that application of neither of the above mentioned techniques 
produces a resulting image having information beyond that available within the supplied 
images. As described above, the generation technique simply results in a new spatial domain 
data set from compositing or manipulating input images. Likewise, the machine vision 
technique simply produces a data set indicative of the position, movement, etc. of an object 
appearing in an input image. 

Additionally, the above mentioned techniques have been isolated in their application. 
For example, image generation techniques typically generate a resulting image by 
mechanically applying a desired generation technique to a selected image or data set. 
Accordingly, chroma-key video generation simply removes an area of an input image having a 
particular color associated therewith and replaces this area with a second input image. 
Likewise, computer generated models and titles merely superimpose the computer generated 
image over an input signal or over a blank screen. As utilized in the prior art, such mechanical 
applications of image generation techniques would benefit little from the application of 
machine vision techniques. 
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Similarly, machine vision techniques are typically applied in order to generate data 
with respect to a object within an image. As such, the applications utilizing machine vision 
techniques are generally disinterested in the manipulation of an output image. Therefore, 
applications of machine vision techniques used in the prior art would not benefit from the 
5 application of image generation techniques. 

However, it may desired to manipulate an image to produce image information beyond 
the two-dimensional spatial domain information available by the above described techniques. 
For example, in stereoscopy, where the sensation of depth obtainable with binocular vision 
due to small differences in parallax producing slightly differing images to each eye of a 
10 viewer, an image more complex than the simple two-dimensional representation is necessary. 
However, the above mentioned isolated digital processing techniques are each insufficient to 
extract information from a digital video image in order to properly manipulate the image to 
produce a resulting stereoscopic image. 

One example of a prior art system is shown in International Application No. 
15 PCT/AU96/00820 filed December 20, 1996, which illustrates how to create 3D images from 
2D input by elongating and/or moving the existing images. This system does not allow for 
objects to be "released" from their background and, when viewed, appear to stick out from the 
screen. 

Therefore, there is a need in the art for a system and method for processing images in 
20 order to extract information beyond that directly ascertainable from an image representation. 

There is a further need in the art for applying information extracted from image 
representations to manipulate the image in order to produce a resulting image representation 
including information beyond that directly available from a source. 

There is a still further need in the art to utilize information extracted from a two- 
25 dimensional spatial domain representation of an image in order to generate an enhanced image 

providing robust images such as a stereoscopic three-dimensional representation of the input 
image. 
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SUMMARY OF THE INVENTION 

These and other objects, features and technical advantages are achieved by a system 
and method which utilizes information with respect to objects within a sequential input image, 
such as is available from the aforementioned machine vision technique, in order to extract, 
5 extrapolate and/or interpolate information about the image not directly presented by a two- 

dimensional spatial domain representation. This information is then utilized with available 
image manipulation techniques to produce a resulting image having robust attributes not 
available through the use of image manipulation techniques alone. 

In a preferred embodiment of the present invention, source images, such as the 
10 aforementioned two-dimensional spatial domain representations, acquired using standard film 
or video capturing techniques are converted into enhanced images having the illusion of three 
dimensions. In an alternative embodiment, the resulting video images are in the form of 
stereoscopic three-dimensional images. Accordingly, the present invention, utilizing machine 
vision techniques, operates to dissect the source image into object components. Through this 
15 process, the present invention is operable to extract or extrapolate information with respect to 
objects, and their interrelationship, as contained within the image. 

Source images may be provided from any number of sources and may include a single 
image or a series of related images. For example, a series of slightly different images which 
may be displayed in rapid temporal sequence in order to create the perception of smooth 

20 motion as, in the case of television or film image sequences, may be used. Utilizing the 
aforementioned machine vision techniques, the present invention operates to interpolate 
spatial domain information with respect to the objects through reference to temporal domain 
information. For example, where an object in the foreground of the spatial domain 
representation moves with respect to other objects throughout the temporal domain, spatial 

25 domain information for any particular image may be interpolated from information available 

in other images in the sequence. The interpolated information is used to fill in the information 
"missing" when each object of the visual image is separated away from the whole. 

Thereafter, the present invention operates to produce a new image utilizing the above 
acquired information. Production of this new image may utilize image manipulation 
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techniques, such as rendering, computer modeling, compositing and overlaying, in order to 
recombine the objects of the source image into a resulting image incorporating the above 
acquired information. 

For example, the present invention may utilize the acquired information to determine 
5 that an image object is in the foreground while another object is in the background. From this 

information, and utilizing the image generation techniques of rendering and overlaying, the 
present invention may operate to produce a new image including object shadowing from a 
heretofore nonexistent light source. 

Likewise, the present invention may utilize this information in combination with the 
10 image generation technique of overlaying to produce two new images wherein the objects of 
each image are overlaid with slight variations. By presenting one image to each of a viewer's 
two eyes, the sensation of depth is obtained due to small differences in parallax. 

It shall therefore be appreciated that a technical advantage of the present invention is 
its ability to extract information beyond that directly ascertainable from a source image. 

15 A further technical advantage is realized in that the information extracted from the 

source image may be utilized to manipulate the image in order to produce a resulting image 
representation including information beyond that directly available from the source image. 

A still further technical advantage of the present invention is its ability to produce a 
stereoscopic three-dimensional image from a source image having only a two-dimensional 
20 spatial domain representation. As such, the present invention is uniquely able to generate 
stereoscopic three-dimensional images from a wealth of pre-existing images. 

Likewise, an additional technical advantage is realized in that the present invention is 
uniquely suitable for producing enhanced images, such as the aforementioned stereoscopic 
images utilizing common, and typically inexpensive, image capturing equipment. 

25 The foregoing has outlined rather broadly the features and technical advantages of the 

present invention in order that the detailed description of the invention that follows may be 
better understood. Additional features and advantages of the invention will be described 
hereinafter which form the subject of the claims of the invention. It should be appreciated by 
those skilled in the art that the conception and the specific embodiment disclosed may be 
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readily utilized as a basis for modifying or designing other structures for carrying out the same 
purposes of the present invention. It should also be realized by those skilled in the art that 
such equivalent constructions do not depart from the spirit and scope of the invention as set 
forth in the appended claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present invention, and the advantages 
thereof, reference is now made to the following descriptions taken in conjunction with the 
accompanying drawings, in which: 

FIGURE 1 illustrates a series of video (or film) image frames; 

FIGURE 2 illustrates the frames of FIGURE 1 dissected into cells each having a single 
object associated therewith; 

FIGURE 3 illustrates the cells of FIGURE 2 having missing object information filled 
in according to the present invention; 

FIGURE 4 illustrates a robust image resulting from a preferred embodiment of the 
present invention; 

FIGURE 5A illustrates an overhead view of physical placement and depth aspects of 
the objects represented in a frame of FIGURE 1 ; 

FIGURE 5B illustrates a robust image representing the object placement and depth 
aspects of FIGURE 5 A resulting from a preferred embodiment of the present invention; 

FIGURE 6 A illustrates an overhead view of the viewing of the physical objects 
represented in a frame of FIGURE 1; 

FIGURE 6B illustrates a set of stereoscopic images representing the object placement 
of FIGURE 6 A resulting from a preferred embodiment of the present invention; 

FIGURE 7 illustrates a block diagram of a preferred embodiment of the present 
invention; and 

FIGURE 8 shows the steps involved in occlusion filling. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Directing attention to FIGURE 1 , a set of captured video (or film) images is depicted 
as frames 101-107. The images represented in these frames may be acquired by any number 
of methods. For example, opto-electronic transducing devices, such as video cameras and 
5 computer scanning devices, may have been used. Likewise, the images might have been 

originally generated on a digital system, such as a general purpose processor-based system or 
personal computer (PC). Moreover, the images may have been produced utilizing a 
combination of the above methods, or the images could have been hand drawn or abstracted 
from film images. The word video herein includes all of the above. 

10 The frames of FIGURE 1 may be representative of a sequence of images in the 

temporal domain. Here, for example, frames 102-106 include a continuity of objects in a 
series of slightly different images which, when displayed in rapid temporal sequence, produce 
a "scene" where the objects appear to move smoothly. 

Video images, such as those depicted in frames 101-107, may be digitized wherein 
15 digital information, (a set of numbers having a direct correspondence to image information in 

the spatial domain), is utilized to store, manipulate, and/or reproduce the images. For 
example, a displayed image such as that of frame 101 might consist of 512 by 640 pixels 
where each pixel is characterized by a range of possible luminous intensities and colors. Each 
of these pixel characterizations includes information representing the image at a particular 
20 point in the spatial domain. 

Accordingly, images of frames 101-107 may be digitized such that the spatial domain 
information is retained for processing, manipulation, storage, and/or presentation by a digital 
processing system. For example, a general purpose processor-based system, such as a PC, 
may accept such a digitized representation in order to process the image according to the 
25 desires of a user. Digital processing may include object identification such as through the use 

of the aforementioned machine vision techniques. Similarly, the PC may process the digitized 
representation in order to manipulate and display the image such as through the use of the 
aforementioned video generation techniques. 
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However, according to a preferred embodiment of the present invention, digital image 
processing includes video processing techniques utilized in a new and novel way so as to 
extract, extrapolate, and/or interpolate new and useful information from a source image. 
Thereafter, a video image is generated incorporating the new information which includes more 
5 than the simple spatial domain data available from the source image. 

Accordingly, a preferred embodiment of the present invention utilizes a general 
purpose processor-based system, such as PC 700 illustrated in FIGURE 7, adapted to 
manipulate video image information through the use of machine vision algorithm 710 and 
video generation algorithm 720. PC 700 includes processor (CPU) 701 coupled to memory 

10 (RAM) 702. RAM 702 provides storage for digitized image information associated with a 

source video image as well as for any video image resulting from the operation of the present 
invention. PC 700 is also adapted to accept input of a source video as well as output a 
resulting video. Of course, acceptance and output of such video may be in digitized form. 
Alternatively, PC 700 may be adapted to accept and/or output analogue video, such as in the 

15 form of National Television System Committee (NTSC) compatible signals. It should be 

noted that while a processor is shown, the system could be hard wires, or could be a series of 
processors. 

PC 700 also includes an operator interface providing information exchange with an 
operator of the system. Such information exchange may include the display of source and/or 
20 resulting video images on a suitable display device. Additionally, the information exchange 
may include an operator selecting and/or inputting information with respect to the generation 
of video images according to the present invention. 

When an image sequence is segmented over time, portions of the desired segment may 
not be defined when they are occluded by other segments. For example, the rectangle segment 
25 120 in FIGURE 1 is not defined when circle 1 1 0 has occluded it as shown in FIGURE 1 , 

Segment 104 and F162, Segment 204C. Fundamental to this invention is the ability to fill in 
this missing data in a realistic manner. 

For the sake of presentation, we will start with some simple cases and build up to the 
general case, as outlined in FIGURE 8. 
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Background 108 in FIGURE 1 represents the easiest case. As to view circle 110 and 
rectangle 120 move across background 108, different areas of the background are exposed. As 
the background does not move, the values for undefined pixels in the first frame can be found 
by looking in the same pixel locations in subsequent frames. This is a direct temporal lookup. 
Note that this lookup can be forward or backward in time. 

Rectangle 120 in FIGURE 1 presents a harder case. As the rectangle is moving, we 
cannot look for a value for the rectangle in the same location at other times - the rectangle was 
not there. For the rectangle, we must determine its motion and then compensate for this 
motion in the temporal lookup. For each pair of subsequent frames, we analyze the motion of 
the data. This can be done by regions (such as MPEG motion estimation), or by pixel, or by 
many other methods. This motion should be analyzed on the segmented image; otherwise, the 
movement of other objects may interfere. For example, if we attempted to analyze the motion 
of the rectangle with the circle still in the frame, we may be thrown off. However, we cannot 
directly analyze the motion of the rectangle when it is not defined. Thus, we interpolate the 
motion of defined areas in order to fully define the motion of the rectangle. This corresponds 
to steps 801 and 803 of FIGURE 8. Optionally, the resultant motion vectors may be 
temporally smoothed in order to define a uniform motion as shown in step 802. 

Once the motion of the rectangle has been determined, we can temporally lookup the 
missing values by searching forward and backward in time along the motion vectors (step 
804). Once the related pixels are found, they can be filtered to generate a value for the 
undefined pixel (step 805). This process is iterated until all undefined pixels are filled. 
Thereafter, some spatial edge cleanup may be employed (as shown in step 806). 

In this example, motion is a transformation between frames or portions of frames. 

In the most general case, transformations can be determined which go beyond motion; 
rotation, translation, scaling, and deformation can all be analyzed. In these cases, the lookup 
of temporally related information becomes more complicated. For example, if scaling 
transformations are found, the current pixel may be related to either a portion of a pixel in 
other frames, or a group of pixels in other frames. In either case, all of the temporally related 
pixels can be used to determine the current value. Consider an object coming towards the 
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screen. This object would have a scaling transformation - it gets bigger with time. An 
undefined pixel at a time t may be temporally related to 8 pixels at time t+5. The 8 pixels 
could then be filtered spatially (i.e., averaged), and combined with other temporally related 
data from other frames in order to create a value at time t. 

Of course, there will be times when no temporal information is available - such as 
when an object is stationary. In these cases, information in key frames is generated using 
other methods (i.e., manually or using spatial mapping and filtering). Once a key frame is 
done, this information will be available to other frames temporally. 

Although a preferred embodiment of the present invention utilizes a general purpose 
processor-based system, it shall be appreciated that any combination of special purpose 
devices and/or general purpose devices adapted to operate according to the present invention 
many be utilized. For example , a general purpose processor-based system may be adapted to 
accomplish particular aspects of the present invention while a special purpose video image 
generation device coupled thereto accomplishes other aspects. 

The machine vision techniques provided by machine vision algorithm 710 are operable 
to differentiate between the objects appearing in a particular source image. For example, 
using these techniques the present invention is able differentiate circle 110 and rectangle 120 
of frames 102-106, from each other, a background, etc. 

The particular method used to distinguish between the objects of an image may be any 
of a number of such methods. In the simplest form, an operator through the use of the 
operator interface may demarcate particular objects of interest such as through identifying a 
subset of spatial domain information identified with particular objects. Alternatively, the 
processor system may distinguish objects within an image through use of a knowledge based 
system such as optional knowledge base 711. Likewise, the processor system may reference a 
series of frames in order to distinguish particular objects through coalescence of spatial 
domain data throughout the series of frames. The latter two methods have the advantage of 
requiring little or no operator input in order to distinguish between the objects. 

From the objects distinguished within the image, particular objects to which, or for 
which, image modification is to be made may be selected. For example an operator may 
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identify particular object images to be manipulated by the present invention. Likewise, the 
present invention may make such a determination based on information such as the relative 
movement of an object from frame to frame. Of course, all of the objects may be utilized for 
image modification, if desired. 

Thereafter, according to a preferred embodiment of the present invention, machine 
vision algorithm 710 tracks the selected objects throughout frames of a scene, such as the 
sequence of frames 102-106. As such, the processor system is utilized to distinguish each of 
the selected objects throughout the series of frames. 

Of course, knowledge based systems or image interpolation may be required when 
tracking images according to the present invention where one of the selected objects is 
occluded by another object. For example, where circle 110 and rectangle 120 of frame 104 are 
being tracked according to the present invention, the processor system may make a 
determination as to the existence and placement of rectangle 120 through reference to the non- 
occluded portions of the rectangle and interpolation of the area there between. Additionally, 
or alternatively, the present invention may track an object in a frame of a series of frames 
through reference to other frames in the sequence. For example, the existence and placement 
of rectangle 120 in frame 104 might be interpolated from the existence and placement of 
rectangle 120 in frames 103 and 105. 

From the information gathered through tracking each of the selected objects through 
each frame of the scene, the images of each frame may be dissected into "cells." The cells of 
the present invention are frame image representations having all information except that 
associated with a particular identified object removed therefrom. Preferably, the cells of the 
present invention are digitized image information sets stored within RAM 702 for utilization 
in manipulating source video image information. Of course, the cells may be displayed or 
output as images, such as through the operator interface of PC 700, if desired. 

Directing attention to FIGURE 2, cell images associated with frames 102-106 of the 
scene illustrated in FIGURE 1 are shown. In each cell of a frame, here cell 202a, 202b, and 
202c being associated with frame 102, includes the objects as tracked throughout the scene. It 
shall be appreciated that cells 202a-206a, although referred to herein as including "objects," 
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do not actually include tracked objects, but rather include the remainder of an image when 
objects of interest have been removed. However, the use of the term objects herein shall 
include such a background image. 

The cells of FIGURE 2 include gaps in the images. For example, cells 202a-206a each 
5 include gaps where the removed objects occluded the background. Likewise, cell 204c 

includes a gap in rectangle 120 where a portion of it was occluded by circle 1 10 in the frame 
image. It shall be appreciated that these gaps are caused by the absence of information due to 
the two-dimensional representation of the image. 

Although not originally available in each individual image of a scene, the present 
10 invention operates to extract information with respect to each object for each cell. The above 
described knowledge based systems or image interpolation techniques may be utilized to fill 
in information missing due to occlusion by an object. Preferably, in order to provide 
information with respect to the missing portions of the cell image, the present invention may 
reference other frames in the sequence where the information of interest is not occluded. For 
15 example, the information missing from the image of cell 203 might be interpolated from that 

of cell 202a and/or 204a. Of course there is no limitation that the missing information be 
interpolated from contiguous cells in the scene. Information from any cell relevant to the 
missing information may be used according to the present invention. 

When an image sequence is segmented over time, portions of the desired segment may 
20 not be defined when they are occluded by other segments. For example, rectangle Segment 
120 in FIGURE 1 is not defined when Circle 110 has occluded it as shown in FIGURE 1, 
Segment 104 and FIGURE 2, Segment 204c. Fundamental to this invention is the ability to 
fill in this missing data in a realistic manner. 

For the sake of presentation, we will start with some simple cases and build up to the 
25 general case, as outlined in FIGURE 8. 

Background 108 in FIGURE 1 represents the easiest case. As the circle 110 and 
rectangle 120 move across background 108, different areas of the background are exposed to 
view. As the background does not move, the values for undefined pixels in the first frame can 
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be found by looking in the same pixel locations in subsequent frames. This is a direct 
temporal lookup. Note that this lookup can be forward or backward in time. 

Rectangle 120 in FIGURE 1 presents a harder case. As the rectangle is moving, we 
can not look for a value for the rectangle in the same location at other times — the rectangle 

5 was not there. For the rectangle, we must determine its motion and then compensate for this 

motion in the temporal lookup. For each pair of subsequent frames, we analyze the motion of 
the data. This can be done by regions (such as MPEG motion estimation), or by pixel, or by 
many other methods. This motion should be analyzed on the segmented image; otherwise the 
movement of other objects may interfere. For example, if we attempted to analyze the motion 

10 of the rectangle with the circle still in the frame, we may be thrown off. However, we cannot 

directly analyze the motion of the rectangle when it is not defined. Thus, we interpolate the 
motion of defined areas in order to fully define the motion of the rectangle. This corresponds 
to steps 801 and 803 of FIGURE 8. Optionally, the resultant motion vectors may be 
temporally smoothed in order to define a uniform motion as shown in step 802. 

15 Once the motion of the rectangle has been determined, we can temporally look up the 

missing values by searching forward and backward in time along the motion vectors (step 
804). Once the related pixels are found, they can be filtered to generate a value for the 
undefined pixel, (step 805). This process is iterated until all undefined pixels are filled. 
Thereafter, some spatial edge cleanup may be employed as shown in step 806. 

20 In this example, motion is a transformation between frames or portions of frames. 

In the most general case, transformations can be determined which go beyond motion: 
rotation, translation, scaling, and deformation can all be analyzed. In these cases, the lookup 
of temporally related information becomes more complicated. For example, if scaling 
transformations are found, the current pixel may be related to either a portion of a pixel in 
25 other frames, or a group of pixels in other frames. In either case, all of the temporally related 

pixels can be used to determine the current value. Consider an object coming towards the 
screen. This object would have a scaling transformation - it gets bigger with time. An 
undefined pixel at time may be temporally related to 8 pixels at time t+5. The 8 pixels could 
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then be filtered spatially (i.e., averaged), and combined with other temporally related data 
from other frames in order to create a value at time t. 

Of course, there will be times when no temporal information is available — such as 
when an object is stationary. In these cases, information in key frames is generated using 
5 other methods (i.e., manually or using spatial mapping and filtering). Once a key frame is 

done, this information will be available to other frames temporally. 

Alternatively, the present invention may operate to interpolate the missing information 
through reference to the information of the particular frame, rather than those of a scene or the 
information may be hand painted or obtained from a source other than the direct input. Once a 

10 portion of the data is entered, the system can use that data to complete the 3D model. Here, 
for example, the processor based system may utilize a knowledge based system in order to 
reference available data and generate compatible fill data therefrom. Where the missing 
information is a part of a homogenous image, such as the background of cells 202a-206a, the 
generation of fill data is obligatory. However, where the image is other than homogenous, the 

15 generation of fill data may require extrapolation of information based upon the image 

information surrounding the missing portion. 

Once the fill data is generated, the objects of the cells may be completed such that each 
cell includes complete information as to the image of each object within the bounds of the 
frame image. Directing attention to FIGURE 3, the cells of FIGURE 2, having the generated 
20 fill data included therein, are illustrated. Here cells 302a-206a, 302b-206b, and 302c-206c 
correspond to cells 202a-206a, 202b-206b, and 202c-206c respectively, which cells in turn 
correspond to frames 102-106 of FIGURE 1. 

It shall be appreciated that by utilizing the information available from the cells of the 
present invention that the image objects may be manipulated, such as through the use of video 
25 generation algorithm 720, to result in a generated video image very different than the source 

image. For example, the objects may be moved with respect to each other and/or the 
background without a causing a hole or gap to appear in the generated image. Likewise, 
objects may be completely removed while still generating a complete image according to the 
present invention. Moreover, the information extracted by the present invention may be 
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utilized to create original frames, such as ones to be placed temporally between those of the 
scene of FIGURE 1 . Here the resulting images may be used to present a scene displaying 
smoother movement of the objects, for example. 

Additionally, the present invention provides sufficient information from which three- 
5 dimensional aspects of the image may be determined and displayed. For example, through the 

information acquired in tracking the objects of the image described above, it is readily 
discemable that rectangle 120 is disposed within in a plane more distant from the vantage 
point than circle 110 when the two objects pass in the scene. This is obvious because circle 
1 10 must be disposed closer to the viewer in order to occlude rectangle 120 as the objects 
10 pass. 

Moreover, having full information with respect to each object throughout each cell, 
and therefore each frame, the present invention may operate to determine three-dimensional 
plane aspects for each object. For example, relative size information of each object from 
frame to frame may be utilized to determine movement of the object into or out of the 
15 foreground throughout the scene. This information taken in conjunction with the relative 

plane information of the objects with respect to each other may be utilized to extrapolate the 
objects' relative plane even where one object is not occluding the other. 

The present invention may utilize the above information in any number of ways. By 
providing depth information in conjunction with the above determined relative plane 
20 information, the image may be given three-dimensional image enhancements. For example, 
through the use of video generation techniques, such as rendering, shading of the image to 
represent shadows cast by the objects may be utilized to create a three-dimensional effect. 

The results of such shading being added to the image of frame 104 is shown in 
FIGURE 4. It shall be appreciated from a review of the image of FIGURE 4, however, that 
25 the provision of depth information alone, although resulting in a three-dimensional optical 
effect considerably more robust than that of the two-dimensional source image, does not 
present as robust of an image as one might actually view. This is so because the objects 
appearing within the image are flat images which are displaced from one another according to 
the depth information described above. 
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Therefore, according to a preferred embodiment of the present invention, in addition to 
depth information, the objects themselves are given depth. This is accomplished through 
video generation techniques such as three-dimensional computer modeling. Here, for 
example, the processor system may generate a three-dimensional wire frame representing the 
full shape of the object. Thereafter, the image information as depicted in the cells of the 
present invention may be wrapped around this wire frame in order to give the object a depth 
aspect. 

Of course, methods other than the above mentioned three-dimensional modeling may 
be utilized according to the present invention in order to give objects a depth aspect. For 
example, shading and highlighting of the object may be accomplished according to a video 
generation technique in order to simulate depth of a particular object. 

Having given individual objects of the cells a depth aspect, the present invention may 
utilize the cell information to provide a robust generated image. Directing attention to 
FIGURE 5 A, an overhead view of circle 110 and rectangle 120 of frame 104 having depth 
aspects is shown. From this view, it can be seen that circle 1 10 is tilted such that the top of 
the circle is closer to the background than is the bottom. Likewise, it can be seen that the 
rectangle remains vertical, but has been wrapped around a wire frame model (i.e., the left side 
of an ovular cylinder). Of course, the aspects of the objects illustrated in FIGURE 5 A are 
arbitrary and may, in fact, include the use of any image manipulation technique. 

Directing attention to FIGURE 5B, an image generated utilizing the robust information 
of the cells as illustrated in FIGURE 5 A is shown. As in the image of FIGURE 4, shading has 
been added to the generated image in order to produce a three-dimensional optical effect. 
However, it shall be appreciated that unlike the image of FIGURE 4, that of FIGURE 5 
includes shading illustrative of the depth aspects of the more robust objects. This shading 
manifests itself as the distortion of the regular shadows of circle 1 10 and rectangle 120 as well 
as the addition of a shadow on the face of rectangle 110 caused by its bending away from the 
light source. 

In addition to the ability to present a more robust two-dimensional image as described 
above, the present invention may also be utilized in the generation of stereoscopic images. 
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Using the information available from the cells of the present invention, the objects may be 
manipulated to result in related sets of generated video images which, when viewed 
simultaneously, give the effect of solidity or depth. For example, the present invention may 
utilize the cell information in combination with the video generation technique of overlaying 
5 to produce two new video images wherein the objects of each image are overlaid with slight 
variations. Utilizing depth information as described above, the present invention may 
determine a proper amount to adjust the placement of each object within each generated image 
in order to create the sensation of depth when the slightly different images are presented to 
each of a viewer's two eyes. This three-dimensional perception is obtained due to small 
10 differences in parallax due to the placement of the objects within the image. 

Directing attention to FIGURE 6 A, an overhead view of how the objects, circle 110 
and rectangle 120, having the above mentioned robust depth information added might 
physically be placed is shown. Also illustrated are projected sight lines associated with the 
left and right eyes of a viewer of the objects. It can be seen from this illustration that there is a 
15 slight displacement associated with the viewer's left and right eyes. This displacement causes 

a difference in parallax as the viewpoint associated with each eye is slightly different. 

Having full object information with respect to each cell, the present invention may 
generate an image associated with the viewpoint of each eye illustrated in FIGURE 6A. Thus, 
two images, providing slightly differing overlap of circle 110 and rectangle 120, may be 
20 generated according to the present invention, such as illustrated in FIGURE 6B. When one 
image each is presented to a viewer's left and right eye, stereoscopic vision is achieved as the 
viewpoint of each eye, including the difference in parallax, is accurately and independently 
reproduced for the associated eye. 

It shall be appreciated that the stereoscopic three-dimensional vision accomplished 
25 according to the present invention is derived from a single two-dimensional source image 

according to the present invention. Of course, although not shown in FIGURE 6B, additional 
robust information, such as the aforementioned shading, may be included in the stereoscopic 
images in order to further enhance the illusion of three-dimensional representation. 
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Although a preferred embodiment of the present invention has been discussed with 
reference to a series, or scene, of source images, it shall be appreciated that an alternative 
embodiment may utilize interpolative techniques and/or knowledge based systems, as 
discussed above, in order to extract missing information with respect to a single frame. 
5 Accordingly, the present invention may be utilized to provide an enhanced image without 

reference to a series of source images. 

Likewise, although the use of a single source image, or series of source images, has 
been discussed herein, it shall be appreciated that the present invention may be utilized with a 
plurality of source images. For example, a source image captured via video camera may be 
1 0 mixed with that of an image generated by a processor based system in order to generate a 
stereoscopic set of images according to the present invention. 

Moreover, it shall be appreciated that, although a preferred embodiment has been 
discussed wherein information with respect to objects is extracted to produce a cell having 
complete object information associated with each object, partial object information may be 
15 utilized. For example, as the information required to provide stereoscopic images having the 
proper parallax typically requires only small amounts of additional information with respect to 
occluded object portions, stereoscopic images may be produced according to the present 
invention without the cells used therefor having complete object information. 

Although the present invention and its advantages have been described in detail, it 
20 should be understood that various changes, substitutions and alterations can be made herein 
without departing from the spirit and scope of the invention as defined by the appended 
claims. 
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WHAT IS CLAIMED IS : 

1. A system for digitally processing an image sequence provided by an image 
source, said system comprising: 

means for differentiating between objects represented in said source images; 

means for parsing said source image sequences according to said differentiated objects 
into image cells, wherein each said image cell includes substantially only information with 
respect to an associated differentiated object; 

means for ascertaining in part from information indirectly ascertained from said 
sequence of source images certain missing information with respect to selected ones of said 
differentiated objects; 

means for supplementing ones of said image cells with at least a portion of said 
ascertained information; and 

means for manipulating ones of said image cells to produce a sequence of resulting 
images enhanced to include said ascertained missing information. 

2. The system of claim 1 wherein said differentiating means includes machine 
vision techniques distinguishing ones of said objects through reference to said source image. 

3. The system of claim 2 wherein said differentiating means comprises: 

means for distinguishing ones of said objects through reference to a plurality of frames 
of said sequence of images. 

4. The system of claim 3 wherein said differentiating means comprises: 

means for recognizing coalescence of image data through ones of said series of image 

frames. 

5. The system of claim 1 wherein said ascertaining means comprises: 

means for extrapolating at least a portion of said ascertained information associated 
with a particular differentiated object of an image frame of said series from another image 
frame of said series. 
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6. The system of claim 1 wherein said ascertaining means comprises: 
means for extrapolating at least a portion of said ascertained information through 

reference to information available from an image frame to which said ascertained information 
relates. 

7. The system of claim 6 wherein said extrapolating means includes reference to a 
knowledge based system. 

8. The system of claim 1 wherein said ascertaining means comprises: 
means for adding at least a portion of external data. 

9. The system of claim 1 wherein said extrapolating means comprises: 
means for adjusting information from another cell. 

10. The system of claim 1 wherein said manipulating means comprises: 
means for utilizing ones of said image cells according to a predetermined image 

generation technique. 

1 1 . The system of claim 10 wherein said predetermined image generation 
technique includes at least one technique selected from the group consisting of: 

chroma-keying; 
image compositing; 
5 image overlaying; 

rendering; 

image transitioning; and 
computer image generation. 

12. The system of claim 10 wherein said resulting image includes three- 
dimensional enhancement of said source image. 
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13. The system of claim 12 wherein said three-dimensional enhancement includes 
ascertained information with respect to a depth aspect of ones of said objects. 

14. The system of claim 12 wherein said three-dimensional enhancement 
comprises a stereoscopic image. 

15. The system of claim 14 wherein said source image is captured without the 
benefit of three-dimensional information. 

16. The system of claim 1 wherein ones of said means are embodied in a general 
purpose processor-based system. 

17. A method for digitally processing an image sequence provided by an image 
source, said method comprising the steps of: 

differentiating between objects represented in said source images; 

parsing said source image sequences according to said differentiated objects into image 
cells, wherein each said image cell includes substantially only information with respect to an 
associated differentiated object; 

ascertaining in part from information indirectly ascertained from said sequence of 
source images certain missing information with respect to selected ones of said differentiated 
objects; 

supplementing ones of said image cells with at least a portion of said ascertained 
information; and 

manipulating ones of said image cells to produce a sequence of resulting images 
enhanced to include said ascertained missing information. 

18. The method of claim 17 wherein said differentiation step comprises the step 

of: 

distinguishing ones of said objects through reference to said source image. 
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19. The method of claim 18 wherein said differentiation step further comprises the 

step of: 

distinguishing ones of said objects through reference to a plurality of frames of said 
sequence of images. 

20. The method of claim 19 wherein coalescence of image data is recognized 
through ones of said series of image frames. 

21. The method of claim 17 wherein said step of ascertaining comprises the step 

of: 

extrapolating at least a portion of said ascertained information associated with a 
particular differentiated object of an image frame of said series from another image frame of 
5 said series. 

22. The method of claim 1 7 wherein at least a portion of said ascertained 
information comprises the further step of: 

extrapolating through reference to information available from an image frame to 
which said ascertained information relates. 

23. The method of claim 22 wherein said extrapolation includes reference to a 
knowledge based system. 

24. The method of claim 17 wherein said step of ascertaining comprises the step 

of: 

adding at least a portion of external data. 

25. The method of claim 17 wherein said step of extrapolating includes the step of: 
adjusting information from another cell. 
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26. The method of claim 17 wherein said step of manipulating comprises the step 

of: 

utilizing ones of said image cells according to a predetermined image generation 
technique. 

27. The method of claim 26 wherein said predetermined image generation 
technique includes at least one technique selected from the group consisting of: 

chroma-keying; 
image compositing; 
image overlaying; 
rendering; 

image transitioning; and 
computer image generation. 

28. A method for processing an input sequence of image representations, said 
method comprising the steps of: 

discriminating among objects represented in said input sequence of image 
representations; 

dissecting said input sequence of image representations into sub-image 
representations, wherein ones of said sub-image representations include information with 
respect to particular ones of said objects; 

ascertaining additional information with respect to said objects included in said sub- 
image representations; 

augmenting ones of said sub-image representations with at least a portion of said 
ascertained information; and 

manipulating ones of said sub-image representations to produce an output sequence of 
image representations. 

29. The method of claim 28 wherein said discriminating step comprises the step 

of: 
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distinguishing ones of said objects through reference to said input sequence of image 
representations. 

30. The method of claim 28 wherein said discriminating step comprises the step 

of: 

distinguishing ones of said objects through reference to a series of image frames. 

3 1 . The method of claim 30 wherein said discriminating step further comprises the 

step of: 

recognizing coalescence of image data through ones of said series of image frames, 
wherein at least a portion of said coalescent data is associated with one of said objects. 

32. The method of claim 28 wherein said input sequence of image representations 
include a plurality of image frames, and wherein said discriminating step comprises the step 
of referencing ones of said plurality of image frames. 

33. The method of claim 32 wherein said ascertaining step comprises the step of: 
extrapolating at least a portion of said ascertained information from said plurality of 

image frames. 

34. The method of claim 28 wherein said ascertaining step comprises the step of: 
extrapolating at least a portion of said ascertained information through reference to 

information available from an image frame to which said ascertained information relates. 

35. The method of claim 34 wherein said extrapolating step comprises the step of: 
referencing a knowledge based system. 

36. The method of claim 28 wherein said manipulating step comprises the step of: 
utilizing a selected image generation technique with ones of said augmented sub- 
image representations. 



WO 00/30039 



27 



PCT/US99/27607 



37. The method of claim 36 wherein said manipulating ones of said sub-image 
representations includes utilizing three-dimensional information. 

38. The method of claim 37 wherein said three-dimensional information comprises 
stereoscopic image information. 

39. The method of claim 38 wherein said input sequential image is captured 
without the benefit of substantial three-dimensional information. 

40. A system for digitally producing an output image representation providing 
improved three-dimensional aspects from a source image having substantially only two- 
dimensional spatial domain information, wherein at least a portion of information with 
respect to the three-dimensional aspects of said output image representation is derived from 

5 said source image, said system comprising: 

a processor-based apparatus having at least a processor and a memory in 
communication with said processor, wherein said processor-based apparatus includes an input 
for accepting an input digitized representation of said source image; 

parsing instructions stored in said memory, for providing a plurality of subsets of said 
10 input digitized representation and wherein each of said subsets include substantially only 

information with respect to a particular object appearing in said source image; and 

means for providing enhancement of certain of said subsets, said enhancement being 
the addition of information missing from said certain subsets. 

41. The system of claim 40 further comprising means for creating a 3D model, 
said 3D model including enhanced ones of said subsets. 

42. The system of claim 40 further comprising means for providing an output 
stereoscopic image representation as a function of select ones of said subsets when operating 
under control of said parsing instructions. 
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43. The system of claim 40 wherein parsing instructions include reference to a 
knowledge base in order to distinguish an object appearing in said source image. 

44. The system of claim 40 wherein said parsing instructions include reference to a 
plurality of image frames provided in said input source image in order to distinguish an object 
appearing in said source image. 

45. The system of claim 45 wherein said reference to said plurality of image 
frames utilizes coalescence of image data through ones of said image frames in distinguishing 
said object. 

46. The system of claim 40 wherein said processor-based apparatus operating 
under control of said parsing instructions operates to embellish upon ones of said subsets to 
include additional information. 

47. The system of claim 46 wherein said additional information substantially 
completes information with respect to an object associated with a particular subset. 

48. The system of claim 46 wherein said additional information is extrapolated at 
least in part from information available within said source image. 

49. The system of claim 46 wherein said additional information is extrapolated at 
least in part from information other than said source image. 

50. The system of claim 40 wherein said processor-based apparatus operating 
under control of said parsing instructions provides information with respect to relative 
placement of said objects within said source image. 

5 1 . The system of claim 50 wherein said last mentioned information is utilized by 
said generation instructions in providing said output image representation. 
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52. The system of claim 40 wherein said processor-based apparatus operating 
under control of said generation instructions operates to produce said output image 
representation at least in part by manipulating ones of said subsets to include a three- 
dimensional aspect. 

53. The system of claim 52 wherein said three-dimensional aspect is selected from 
the group consisting of: 

shadowing; 
highlighting; and 
three-dimensional modeling. 

54. The system of claim 52 wherein said manipulating includes the use of a video 
generation technique selected from the group consisting of: 

chroma-key; 
rendering; 
compositing; 
overlaying; and 
modeling. 

55. The system of claim 40 wherein said output image representation comprises a 
stereoscopic set of images. 

56. A system for digitally producing a stereoscopic image pair from a two- 
dimensional image, wherein at least a portion of parallax information with respect to said 
stereoscopic image pair is derived from said two-dimensional image, said system comprising: 

an input for accepting said two-dimensional image and separating object images 
provided within said two-dimensional image into discrete image cells, wherein said input 
accepts a scene of related two-dimensional images and tracks said object images throughout 
said scene in order to separate object images provided within ones of said two-dimensional 
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images into said discrete image cells, said input also supplementing said object images of said 
discrete image cells based on said scene of related images; and 

an output for generating said stereoscopic image pair from ones of said discrete image 
cells wherein said stereoscopic image pair includes at least portions of said object images as 
supplemented by said input. 

57. The system of claim 56 wherein said input supplements ones of said object 
images through reference to other ones of said two-dimensional images. 

58. The system of claim 56 wherein said output utilizes generation means to 
enhance said stereoscopic image. 

59. The system of claim 58 wherein said enhancement includes image 
manipulation selected from the group consisting of: 

shading; and 
highlighting. 

60. The system of claim 56 wherein said generation means include at least one 
generation means selected from the group consisting of: 

overlay; 

chroma key; 

rendering; 

compositing; 

computer generation; 

look forward processing; and 

look backward processing. 

61 . A method of converting an input sequence of 2D images into a 3D 
representation, said method comprising the steps of: 

segmenting said input sequence of images into a plurality of image sequences; 
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supplementing said segmented image sequences with information derived from said 
input sequence of images; and 

providing a 3D model for certain of said sequences of images, said 3D model 
including said supplemented image sequences. 

62. The method of claim 61 further including the step of rendering stereoscopic 
views of said 3D models. 

63. The method of claim 61 wherein said supplementing step includes the steps of: 
processing information from other frames of said input sequence; and 

correcting said other frame information in accordance with a set of established rules. 

64. The method of claim 61 wherein said information is supplemented from other 
information within said input sequence for missing information and wherein said information 
is missing as a result of said segmentation. 

65. The method of claim 64 wherein said information is supplemented by at least 
one of the techniques selected from the following list: 

information added from an external service; 
information from a look- ahead set of sequences; 
information from a look-backward set of sequences; 
information from a look-up table; and 

information processed at least in part from one or more of the above cited techniques. 

66. A system for converting an input sequence of 2D images into a 3D 
representation, said system comprising: 

means for segmenting said input sequence of images into a plurality of image 
sequences; 

means for supplementing said segmented image sequences with information derived 
from said input sequence of images; and 
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means for providing a 3D model for certain of said sequences of images, said 3D 
model including said supplemented image sequences. 

67. The system of claim 66 wherein said system includes rendering stereoscopic 
views of said 3D models. 

68. The system of claim 66 wherein said supplementary means further comprises: 
means for processing information from other frames of said input sequence; and 
means for correcting said other frame information in accordance with a set of 

established rules. 

69. The system of claim 66 wherein said information is supplemented from other 
information within said input sequence for missing information, wherein said information is 
missing as a result of said segmentation. 

70. The system of claim 69 wherein said information is supplemented by at least 
one of the techniques selected from the following list: 

information added from an external service; 
information from a look- ahead set of sequences; 
5 information from a look-backward set of sequences; 

information from a look-up table; and 

information processed at least in part from one or more of the above cited techniques. 
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